DISTRIBUTION  STATEMENT  A.  Approved  for  public  release;  distribution  is  unlimited. 


Optimized  Infrastructure  for  the  Earth  System  Prediction  Capability 

PI:  Cecelia  DeLuca 
325  Broadway 
Boulder,  CO  80305 

phone:  (303)  497-3604  email:  Cecelia.Deluca@noaa.gov 

Award  Number:  N00014-13-1-0508 
http://www.earthsystemcog.org/projects/espc-infrastructure/ 


LONG-TERM  GOALS 

Note:  This  project  ends  on  October  3 1 ,  20 14.  Ongoing  work  will  be  reported  through  the  project  An 
Integration  and  Evaluation  Framework  for  ESPC  Coupled  Models  (PI  Kirtman). 

The  Earth  System  Prediction  Capability  (ESPC)  interagency  program  was  established  in  2010  as  an 
effort  to  improve  collaboration  across  the  federally  sponsored  environmental  research  and  operational 
prediction  communities.  The  ESPC  goal  is  to  coordinate  scientific  development  and  operational 
implementation  of  improved  global  prediction  at  the  weather  to  climate  interface.  Part  of  the  ESPC 
role  is  to  advocate  for  and  support  the  basic  foundations  of  modeling  and  data  systems,  such  as  the 
computational  efficiency  of  models  and  common  coupled  system  architectures. 

The  Optimized  Infrastructure  for  ESPC  (01  for  ESPC)  project  is  working  to  advance  the  computational 
foundations  of  ESPC  through  three  integrative  activities.  These  activities  are  designed  to  leverage 
numerical  libraries  across  agencies;  to  explore  the  impacts  of  emerging  computing  platforms  on 
coupled  model  infrastructure;  and  to  reconcile  conventions  across  different  model  architectures.  The 
activities  are  serving  as  a  starting  point  for  a  broader  effort  funded  through  the  2013  NOPP  program 
Advancing  Air-Ocean-Land-Ice  Global  Coupled  Prediction  on  Emerging  Computational  Architectures 
(AOLI). 

OBJECTIVES 

1.  Update  Numerical  Libraries  in  ESMF.  The  Earth  System  Modeling  Framework  (ESMF)  is 
widely  used  in  the  ESPC  community  for  building  and  coupling  models.  We  will  adapt 
MOAB,  a  finite  element  mesh  package  in  active  development  at  the  University  of  Wisconsin 
and  Argonne  National  Laboratory,  for  use  as  an  ESMF  computational  kernel.  The  intent  is  to 
evaluate  MOAB  as  a  replacement  for  the  native  ESMF  implementation  of  finite  element  mesh 
software.  The  framework  uses  this  mesh  software  to  create  a  unified  representation  of 
structured  and  unstructured  grids,  and  define  operations  that  work  across  grid  types.  The 
desired  outcome  is  an  immediate  boost  of  new  features  in  ESMF,  and  in  the  longer  term,  an 
opportunity  to  leverage  the  work  that  goes  into  ongoing  functions  and  optimizations  in  MOAB 
and  associated  DOE  libraries. 
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2.  Optimize  Component  Architecture  for  GPUs.  We  will  prototype  a  set  of  mechanisms  that 
enable  a  component  framework  to  optimize  for  accelerators,  and  explore  how  accelerators  may 
affect  mapping  of  components  to  resources.  The  product  will  be  a  code  consisting  of  sample  or 
toy  components  with  representative  operations.  The  intent  of  this  research  is  to  infonn  both 
infrastructure  development  and  optimizations  undertaken  by  ESPC  modeling  teams. 

3.  Couple  HYCOM  to  CESM.  We  will  work  with  developers  of  the  HYbrid  Coordinate  Ocean 
Model  (HYCOM)  to  couple  it  to  the  Community  Earth  System  Model  (CESM).  This  will 
enable  HYCOM  to  be  evaluated  in  many  new  coupled  configurations.  The  activity  will  also 
define  a  mapping  between  the  different  coupling  rules  and  conventions  implemented  in 
HYCOM  and  CESM.  HYCOM  uses  a  standard  ESMF-based  interface  developed  by  a  NOAA 
and  DoD  consortium  (the  National  Unified  Operational  Prediction  Capability,  or  NUOPC 
conventions),  while  CESM  has  its  own  native  format  based  on  the  Model  Coupling  Toolkit 
(MCT).  This  is  an  important  exercise  because  that  mapping  will  apply  not  just  to  this  coupling, 
but  will  provide  insight  into  how  any  components  that  use  the  NOAA/DoD  interface  can  be 
coupled  into  a  CESM-based  system,  and  vice  versa. 

APPROACH 

Team  members  hold  weekly  telecons  to  discuss  plans  and  progress.  Participants  include  both  technical 

and  scientific  team  members,  below: 

Argonne  National  Laboratory/University  of  Chicago/University  of  Wisconsin:  Rob  Jacob,  Jayesh 
Krishna,  Tim  Tautges,  Jane  Hu 

Florida  State  University:  Alexandra  Bozec,  Eric  Chassignet 

NCAR:  Jim  Edwards,  Mariana  Vertenstein,  Tony  Craig 

NASA  Jet  Propulsion  Laboratory:  Peggy  Li 

Naval  Research  Laboratory:  Alan.Wallcraft 

University  of  CA  San  Diego:  Caroline  Papadopoulos,  Julie  McClean 

University  of  Colorado/NOAA:  Cecelia  DeLuca,  Kathy  Saint,  Gerhard  Theurich,  Fei  Liu,  Bob 
Oehmke,  Walter  Spector 

University  of  Miami:  Ben  Kirtman 


Above  are  subscribed  to  an  espc_tech@list.woc.noaa.gov  mailing  list  used  for  organizing  the  calls. 

A  website  for  the  01  for  ESPC  project  was  created  on  the  CoG  collaboration  environment: 
http://www.earthsystemcog.org/projects/espc-infrastructure/ 

Some  of  the  01  for  ESPC  project  activities  are  closely  associated  with  a  Coupling  Testbed,  which  is  a 
collaborative  area  for  groups  working  on  research  problems  involving  model  coupling.  The  Coupling 
Testbed  was  funded  under  the  larger  Integration  and  Evaluation  Framework  NOPP  award.  A  site  for 
the  Coupling  Testbed  was  also  initiated  and  is  here: 
http://www.earthsystemcog.org/projects/couplingtestbed/ 
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Initial  roadmaps  were  created  for  each  of  the  three  objectives,  and  are  linked  under  the  Roadmap  tabs 
on  the  two  sites  above.  The  roadmaps  include  status  at  start,  team  members,  desired  outcomes,  and  a 
sequence  of  tasks. 

1 .  Update  Numerical  Libraries  in  ESMF 

http://www.earthsystemcog.org/projects/espc-infrastructure/moab_in_esmf 

2.  Optimize  Component  Architectures  for  GPUs 

http://www.earthsystemcog.org/projects/couplingtestbed/acceleratorplans 

3.  Couple  HYCOM  to  CESM 

(Part  1)  Introducing  NU OP C  conventions  into  CESM 
http://www.earthsystemcog.org/projects/espc-infrastructure/nuopc_in_cesm 
(Part  2)  HYCOM  in  CESM  scientific  validation 

http://www.earthsystemcog.org/projects/espc-infrastructure/hycom_in_cesm_science 

The  roadmaps  are  being  updated  by  team  members  prior  to  the  weekly  telecons,  and  progress  and 
issues  are  discussed  during  the  calls.  Where  appropriate,  test  reports  describing  how  a  code  base  was 
validated  are  linked  to  the  roadmaps.  These  test  reports  include  information  that  supports  the 
reproducibility  of  results. 

WORK  COMPLETED 

The  Roadmap  pages  listed  under  Approach  contain  detailed  descriptions  of  the  work  completed.  See 
summaries  below. 

Update  Numerical  Libraries  in  ESMF 

In  July,  2014  the  MOAB  team  (Wilson,  Tautges,  Hu)  completed  the  capability  to  migrate  a  MOAB 
mesh  to  a  new  distribution.  This  is  required  for  MOAB  to  be  used  as  a  basis  for  ESMF  grid  remapping 
and  related  functions.  The  MOAB  mesh  migrate  code  can  take  an  input  mesh  and  a  specification  of  the 
processor  destinations  and  create  a  new  mesh  on  those  processors.  In  addition,  the  MOAB  team  added 
options  to  move  variable  data  stored  on  the  mesh  and  to  delete  the  original  mesh.  The  ESMF  team  is 
using  a  set  of  test  cases  that  they  developed  to  validate  the  new  capabilities. 

Last  year,  the  ESMF  team  (Spector)  added  the  MOAB  library  to  the  ESMF  source  code  and  build 
process.  This  year,  Bob  Oehmke  added  a  switch  that  can  select  either  MOAB  or  the  original  ESMF 
mesh  implementation.  Originally  the  plan  was  to  allow  a  swap  during  compile  time.  However,  it  was 
recognized  that  this  would  make  testing  difficult  as  every  capability  not  yet  supported  would  break. 
Instead,  a  runtime  method  for  swapping  was  chosen,  allowing  particular  tests  to  be  changed  to  work 
with  MOAB.  The  next  stage  is  to  allow  the  creation  of  an  ESMF  Mesh  with  a  MOAB  mesh 
underneath.  During  August  and  July  2014,  Bob  experimented  with  MOAB  mesh  creation  and  data 
handling  capabilities  to  prepare  for  this  step.  A  partial  version  of  this  mesh  creation  capability  was 
implemented  in  Sept.  2014.  The  code  can  successfully  create  the  vertices  (i.e.  the  representations  of 
point  locations)  of  a  MOAB  mesh  from  the  ESMF  specification.  The  implementation  of  the  code  to  tie 
these  vertices  into  the  complete  mesh  has  been  partially  completed,  but  still  needs  to  be  finished. 
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Optimize  Component  Architectures  for  GPUs 

A  set  of  prototype  applications  exploring  the  possibilities  of  using  accelerators  with  a  coupling 
framework  was  completed  by  Jayesh  Krishna  of  ANL  and  will  be  reported  under  the  NOPP  award 
An  Integration  and  Evaluation  Framework  for  ESPC  Coupled  Models.  Gerhard  Theurich  of  the 
ESMF  team  advised  on  the  ESMF  implementation  and  design  strategies. 

Couple  HYCOM  to  CESM 

The  integration  of  the  FIYbrid  Coordinate  Ocean  Model  (FIYCOM)  with  the  Community  Earth  System 
Model  (CESM)  involves  three  primary  tasks:  create  a  National  Unified  Operational  Prediction 
Capability  (NUOPC)  interface  or  "cap"  for  the  latest  HYCOM  model  (Theurich  and  others);  modify 
CESM  to  implement  the  NUOPC  framework  (Vertenstein  and  Edwards);  and  integrate  the  HYCOM 
NUOPC  cap  (and  model)  into  the  NUOPC-CESM  system  as  an  alternate  ocean  model  (Saint,  Bozec, 
Liu,  and  others). 

Originally,  a  NUOPC  cap  for  HYCOM  rev37047  was  created  an  integrated  into  the  CESM  baseline 
and  was  able  to  run  standalone  using  the  NUOPC  Driver  in  CESM.  More  recently,  an  updated 
NUOPC  cap  for  HYCOM  rev45185,  replaced  the  original  HYCOM  in  CESM,  and  now  includes  the 
ability  to  import  and  export  data  using  Earth  System  Modeling  Framework  (ESMF)  Fields  and  Grids. 

The  integration  of  the  NUOPC  framework  into  CESM  is  an  ongoing  process  that  involves  several 
stages  of  implementation.  Last  year,  Mariana  Vertenstein  and  Jim  Edwards  of  the  CESM  team 
restructured  the  CESM  coupler  and  driver  so  that  it  can  accommodate  NUOPC  components.  CESM 
can  now  support  a  NUOPC  Driver  driving  the  system,  which  involves  the  coupling  of  all  7  model 
types  (atm,  ocn,  ice,  lnd,  rof,  glc  and  wav)  with  a  NUOPC  Mediator.  While  the  Mediator  provides 
external  interfaces  and  behavioral  rules,  it  is  using  the  existing  coupling  code  from  CESM  to 
implement  its  internal  functionality.  CESM  handles  multiple  variations  of  these  component  types, 
with  dead,  stub,  data  and  active  versions.  NUOPC  Models  are  used  to  "cap"  these  components,  and 
there  are  currently  NUOPC  caps  for  all  of  the  data  components  (datm,  docn,  dice,  dlnd  and  drof),  3  of 
the  stub  components  (slnd,  sglc  and  swav),  and  3  of  the  active  components  (cice,  pop2  and  hycom). 

An  initial  test  of  the  NUOPC-CESM  version  was  to  run  the  Parallel  Ocean  Program  (POP2)  ocean 
with  the  CICE  model  and  ensure  that  results  were  the  same  as  in  native  CESM,  which  uses  the  Model 
Coupling  Toolkit  (MCT).  Fei  Liu  completed  this  task,  which  used  NUOPC  caps  that  he  had  created 
for  the  POP2  ocean  and  CICE  ice  components.  The  test  configuration  was  a  CESM  G  compset  with 
POP2  and  CICE  as  active  components.  The  NUOPC  version  of  the  model  produced  bitwise  identical 
results  with  the  native  CESM  version.  A  test  report  of  this  task  is  available  at 
https://www.earthsystemcog.org/projects/espc-infrastructure/cice-pop2-test-report-14091 1 

Kathy  Saint  led  the  task  of  HYCOM  integration  with  CESM.  An  initial  step  was  to  ensure  that  a 
NUOPC-CESM  1.3  beta04  case  that  includes  HYCOM  as  the  ocean  component  (instead  of  POP2)  and 
CICE  as  the  sea  ice  component  validates  in  a  simple  CESM  compset  C  configuration.  This 
configuration  produced  bitwise  reproducible  results  with  HYCOM  running  standalone,  compared  with 
HYCOM  running  outside  of  CESM,  see  test  report:  https://www.earthsystemcog.org/projects/espc- 
infrastructure/hyc  omruntestrpt 

The  ESMF  team  is  in  the  process  of  implementing  Field  transfers  using  the  NUOPC  software,  while 
continuing  to  use  the  CESM  software  to  handle  the  custom  coupling  code  (such  as  flux  calculations). 

A  next  critical  step  is  to  validate  HYCOM  running  in  a  CESM  compset  G  configuration,  with  active 
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HYCOM  and  CICE.  Currently  one  field,  sea  surface  temperature,  is  being  exported  from  HYCOM  in 
this  configuration,  and  the  team  is  in  the  process  of  validating  that  it  is  being  transferred  correctly. 

In  addition,  ESMF  team  members,  Alexandra  Bozec  and  others  are  going  through  the  list  of  fields 
imported  to  and  exported  from  HYCOM,  and  detennining  which  of  the  HYCOM  import  and  export 
fields  correspond  to  those  in  the  CESM  system.  We  have  all  of  the  export  fields  associated,  and  4 
import  fields  remaining. 

Additional  test  reports  include: 

o  All  data  components  using  ESMF  Fields:  https://www.earthsystemcog.org/projects/espc- 
infrastructure/datamodelfieldstestrpt 

o  NUOPC  POP2  running  with  other  data  components: 

https://www.earthsystemcog.org/projects/espc-infrastructure/pop_only_run_test_rpt 

RESULTS 

The  major  results  for  this  year  include  the  capability  to  switch  between  MOAB  and  the  native  ESMF 
finite  element  mesh,  and  the  ability  to  partially  represent  an  ESMF  mesh  object  using  MOAB.  The 
CESM-HYCOM  coupling  has  resulted  in  a  redesign  of  the  CESM  coupling  infrastructure  to 
accommodate  NUOPC  interfaces.  Initial  configurations  using  NUOPC-CESM  have  been  validated. 

Test  reports  were  written  for  a  NUOPC-CESM  configuration  with  active  POP2  ocean  and  CICE  vs  the 
native  non-NUOPC  version,  and  a  NUOPC-CESM  standalone  version  of  HYCOM  vs  the  non-NUOPC 
version.  Current  work  focuses  on  validating  a  coupled  configuration  of  NUOPC-CESM  with  active 
HYCOM  and  CICE.  In  addition,  a  set  of  applications  was  developed  that  explore  the  possibilities  of 
using  accelerator  technology  in  the  context  of  a  framework  for  coupled  modeling.  This  work  will  be 
summarized  under  the  annual  report  for  the  project  An  Integration  and  Evaluation  Framework  for 
ESPC  Coupled  Models. 

IMPACT/APPLICATIONS 

The  results  obtained  here  can  be  viewed  as  a  head  start  for  the  linked  proposals  funded  under  the  2013 
NOPP  AO  LI  awards.  There  is  a  direct  correspondence  between  this  preliminary  work  and  the  AO  LI 
project  An  Integration  and  Evaluation  Framework  for  ESPC  Coupled  Models  (PI:  Ben  Kirtman).  The 
Kirtman-led  project  also  focuses  on  a  coupled  CESM-HYCOM  configuration  and  builds  on  joint 
technical  work  involving  NCAR,  NRL,  DOE  laboratories,  and  ESMF/NUOPC  developers. 

The  work  presented  here  is  likely  to  inform  the  technical  strategies  adopted  by  other  NOPP  projects  as 
well.  For  example,  streamlining  the  CESM  coupler  while  introducing  NUOPC  code  is  enabling  ESMF 
data  structures  and  methods  that  support  in-memory,  during-run  grid  remapping  to  be  introduced  into 
CESM  in  an  efficient  way,  preserving  existing  infrastructure  where  it  makes  sense.  This  capability  is 
likely  to  be  a  first  step  in  coupling  components  with  adaptive  grids,  an  objective  of  the  NPS-NRL- 
Rice-UIUC  NOPP  AOLI  project. 
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RELATED  PROJECTS 


The  01 for  ESPC  project  is  most  closely  related  to  the  projects  supported  under  the  2013  NOPP  AOLI 
awards,  listed  on  this  page:  http://coaps.fsu.edu/aoli/projects 

The  current  work  is  particularly  relevant  to  the  AOLI  project  An  Integration  and  Evaluation 
Framework  for  ESPC  Coupled  Models  (PI:  Ben  Kirtman),  which  also  focuses  on  a  CESM-HYCOM 
configuration  and  includes  joint  technical  work  involving  NCAR,  DOE  laboratories,  and 
ESMF/NUOPC  developers. 
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